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ABSTRACT 


By  invoking  the  ergodic  theorem,  most  of  the  important  averages  of  prob¬ 
abilistic  systems  that  can  be  represented  as  semi -Markoff  systems  can  be  ob¬ 
tained  as  limits  of  ratios,  precisely  as  in  the  cases  where  the  law  of  large  num¬ 
bers  is  applicable.  The  procedure  ia  illustrated  with  several  examples. 


This  paper  is  published  as  a  Research  Contribution  to  provide  as  early 
as  possible  the  results  of  CNA  supoorting  analysis  to  the  Naval,  Marine, 
and  analytical  communiiy. 
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SEMI-MARKOFF  PROCESSES 


1.  There  has  recently  been  some  interest  in  the  application  of 
the  theory  of  semi-Markoff  processes  to  problems  in  ASW.  The 
purpose  of  this  memorandum  is  to  present  some  simple  proofs  of 
the  main  results. 

2.  Suppose  we  are  studying  a  system  with  a  finite  number  of 
states  1,  2,  . ..,  K,  and  that  the  system  changes  its  state  ac¬ 
cording  to  a  given  Markoff  structure,  with  constant  transition 

matrix  A  =  {a. if  x_  and  x  .  denote  the  states  at  the  n 

th  n  n+-L 

and  (n+1)  stages,  we  have 

PrCxn+l  "  jl*„  =  ^  '  aij- 

We  suppose  that  there  is  a  unique  vector  <tt^,  . . . ,  n^ 

with  strictly  positive  components,  and  ^  tt^  =  1,  for  which 

i 

(t^,  ...,  rrK)  A  =  (tt^,  ... ,  ttk).  The  tt^  are  the  limit¬ 
ing*  or  steady-state,  probabilities.  We  also  suppose  that  aii=  0 

for  i  =  1,  ...,  K.  This  structure  is  fairly  common  in  ASW  models, 
wherein  the  states  refer  to  search,  being  in  contact,  localizing, 
etc. 


3.  We  now  suppose  also  that  we  are  given  waiting-time  densities 
(w)  ,  such  that  if  a  transition  from  i  to  j  will  take  place,  the 

random  variable  w  is  chosen  from  cp.^,  and  the  system  stays  in 

state  i  for  w  units  of  time.  For  simplicity,  we  assume  that  w  can 
take  on  the  values  1,2,3,...  only  (e.g.,  minutes),  but  there  is 
no  difficulty  in  extending  the  results  to  continuous  time.  (In 
some  applications,  the  cp^  depend  only  on  i,  not  on  j.)  If  we 

now  regard  the  time-history  xfc  of  states,  t=0,l,2,...,  we  have  a 


random  process,  but  it  is  not  Markoff;  the  conditional  probability 
of  xfc+^  given  the  past  is  not  the  same  as  that  given  only  x^.  We 
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need  to  know  how  long  the  system  has  been  in  its  present  state. 
Such  a  structure  as  this  is  called  a  semi-Markoff  process. 


4.  The  way  to  proceed  is  clear.  We  take  as  the  state-space,  the 
set  of  all  (i,r),  i»l,2,...,K;  y=I,2,...,  in  which  the  first  coor¬ 
dinate  i  is  one  of  our  original  states,  and  t  denotes  the  time 
elapsed  from  the  instant  the  system  entered  i  to  the  present  time. 
For  exz  pie,  suppose  the  first  few  i's  are  i=2,6,3,l,  with  waiting 
times  \  *3,5,  4.  The  graph  of  i  versus  t  is  as  follows: 


i 


The  states  (i,T)  are:  (2,1),  (2,2),  (2,3),  (6,1),  (6,2),  (6,3), 
(6,4),  (6,5),  (3,1),  (3,2),  (3,3),  (3,4),  ... 


THE  ERGODIC  THEOREM 


5.  It  is  easy  to  see  that  the  randan  process  (it,rt)  is  a  Markoff 
process,  for  all  the  information  contained  in  the  past  (i^,T^), 

...  (in#Tn)  is  already  contained  in  (in»Tn),  as  far  as  the  con¬ 
ditional  probability  of  (in+1,Tn+1)  concerned.  Moreover,  be¬ 
cause  of  our  assumption  on  the  matrix  A  *  £a^j } ,  it  also  follows 

that  our  present  structure  is  irreducible,  and  every  state  (i , t) 
is  recurrent.  Therefore,  if  the  (unique)  limiting  probabilities 


a. 
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are  used  as  the  initial  probabilities  on  (i,T)  for  t  =  0,  our 
process  (it,Tt)  will  be  a  stationary  process,  and  the  "ergodic 

hypothesis"  will  be  valid^  ensemble  averages  and  time  averages 
are  equal. 

6.  More  precisely,  let  fi  denote  the  space  of  all  sample  functions 
(it/Tt)/  t  =  0,±  1,±2,...  (extending  to  the  past  as  well  as  the 

future) .  Let  F(uj)  denote  a  function  defined  for  each  sample 
function  u) .  Then 

J  F<«r)dP(«f)  =  (1) 

a 


for  almost  every  u >.  Here  P  denotes  the  probability  in  0, 
and  Us  is  the  sample  function  obtained  by  shifting  u>  one  time 
step  to  the  left.  As  an  example,  suppose  f(i,T)  is  a  given 
function  of  i  and  t.  Define  F(iu)  on  0  as  follows:  If 

®  ^ ...  ( >  ( ig  /  tq)  ,  ( i^ ,  ■  ,  (  ij ,  i  ...  ^  , 

then  put 

F(u>)  =  f(i0,T0). 


Since 


Jf(oj)  dP(u)) 


E[f(i,T)],  we  have  from  (1)  , 


E[f]  =  ^  +  f(il'Tl}  +  •••  +  f<i: 


n'T»>]/n 


+  1 


(2) 


7.  For  the  applications,  we  are  usually  interested  in  the  lim¬ 
iting  value  as  t  -  *  of  some  probability  or  expected  value 
associated  with  our  process  ( it , Tt) .  This  limiting  value,  or 

steady-state  value  (which  is  independent  of  the  starting  position 
of  (i,T))  is  the  same  as  what  we  would  obtain  for  any  value  of  t 
if  we  started  our  process  with  random  values  of  (i,r)  whose  prob¬ 
abilities  were  equal  to  the  limit  probabilities  of  (i,T)  as 
t  ®.  This  implies  that  our  answer  is  obtainable  in  0,  in  the 

form  j  F(u))  dP(uj)  ,  and  then  (1)  can  be  invoked. 

n 
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B  • 


f-  ' 


EXAMPLES 


8.  We  begin  with  the  limiting  values 


°i,T  "  t~Pr[(it''rt>  *  <i'T>] 


(3) 


For  this  purpose,  we  define  F(u>)in  0: 

0,  ±  1,  y  2,. 


for  (»  *  {<  iv  %>  j  v 


•}. 


F(w) 


1  if  (W 


(i#T) 

0  if  (iQ.T^  4  U,T) 


(4) 


Then 


°i'T 


J  F(tu)dP(u>) 
n 


lim 

n-“ 


V  Sl+S2"»+Sn 


n  +  1 


(5) 


in  which 


1 

0 


if  (iy,Tv)  «  (i,*) 


otherwise. 


Let  us  start  with  (i0,x0)  *  (1,1) ,  and  let  our  sample  function 
be  of  the  fora 


(1,1) ,  (1,2)  ,...,(1, w^) ,  ( i~2 ,1) ,  (i-2 * ^  , » •  *  ( I2 '^2^  * 

....(1^,1),  (i^,2) , . . .  (1]t*w]c) » •  •• 


with 


n  «  w1+w2+...+w^  .  There  are  k  possibly  different 


i  *i  in  this  segment  0  <  t  s  n,  and  each  is  repeated  w^  times. 
We  will  have  roughly  krr^  of  them  equal  to  i  (cf.  paragraph  2); 
for  each  i^  that  equals  i,  we  will  have  exactly  one  tv  *  t  if 
and  only  if  w#  ^  t,  and  none  if  wm  <  t.  Now,  given  that  im  *  i, 
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} 


»^yfiravi^-n 


the  waiting  time  density  in  state  i  is 


^(w) 


-E 

j 


*i  j (w)  aij 


(6) 


So  the  probability  that  t  £  wm,  given  i  =  i. 


is 


•  def 

W'=T 


(7) 


It  follows  that  roughly  Ictt^E^(t)  of  the  §'s  in  (5)  equal  1,  and 
the  quotient  in  (5)  is  about 


ktr  ±Ei  (  t) 
n 


TTjEj(T) 


(Wi+  w2  + 


+  wk)  A 


(8) 


:Cn  the  denominator ,  we  have  in  the  sum  w^  +  ...  +  about 
kn^  w*  8  from  q>^(w) ,  kn2  w's  from  <p2(w)  ,  etc.  bet 

CD 

£  «p±  (w)  vr  , 

w=l 

the  mean  wait-time  in  state  i.  Then  we  have 


W1  + 


+  w^  **  krr^  +:  kTT2w2+ 


kTTKwK 


and  (5)  and  (8)  now  give 


IT 


"iEi<T> 

TT.W.  +  ...  +  TT. 


9 
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i«6«  f 


i,  t 


TT^T) 

2ni*i 


(9) 


This  is  an  important  result. 


9.  We  may  also  require 


o.  =  Pr 

i  t*® 


[v  0 


(10) 


Evidently,  we  have 


a,  *  V  a. 

i  a-  i,t 


We  use  (9)  and  observe  that 


00 

^  Ei(T)  *  £  wcPi^  =  *i 


T=»l 


W=1 


(ID 


SO  that 


IiZl 


1 


(12) 


This  is  also  important.  Note  the  difference  between  tt^  and 


10.  In  a  forthcoming  memo,  O.  Culbertson  defines  his  states 
not  as  we  do,  but  by  (i,y) ,  in  which  i  has  the  same  meaning  as 
ours,  but  y  denotes  the  time  remaining  in  state  i  until  the 
next  jump.  Using  our  set-up,  take  y  fixed  and  i  fixed,  and  de 
fine  F(u>)  in  0  thus:  if 
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i 


in  which  is  the  waiting-time  for  the  currently  occupied  state. 

Now,  given  that  i  =  i,  let  w  denote  the  waiting  time  for  the 

current  sojourn  in  this  state.  Then  the  t’s  that  we  would  observe 
are  r  =  l,2,...w  while  the  y's  are  y  =  w,  w~l,...,l;  i.e. ,  ex¬ 
actly  one  of  the  y's  will  equal  our  specified  y  if  and  only  if 
one  of  the  t's  does.  Therefore  the  argument  given  in  paragraph 
8.  holds  here  also,  and  we  get 


a . 

i/y 


TT.E.  (y) 


(15) 
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V"  ■  a®«SaWfflg«rww;TlS^ 


the  (1,t) *s  and  (i,y)‘s  have  exactly  the  same  distribution. 

11.  Another  question  that  is  often  of  interest  concerns  the  mean  re¬ 
turn  time  to  state  1,  given  that  the  system  has  just  left  state  1. 

By  examining  a  few  typical  sample  functions  (and  their  graphs) , 
we  quickly  see  that 

E  [return  time  to  state  1  |  system  just  left  state  1] 
j  F  dP 

*  ~r2 -  (16) 

G  dP 


in  which 


F(i») 


^smallest  positive  t  for  which  i.  *  1  if  in  =1 

L  t  u 

I  &  i.*  1 

[o  otherwise 


II  if  in  *  1  &  i.  *  1 
0  otherwise  . 


By  (1) ,  we  obtain 


t«  *4*  e  #  »  •  t 


E  [return  time  |  just  left]  *  ,  ,  .  2...  — --A-— 

I**  *  $2  *  •  •  •  •  *3j^ 


in  which  the  ty  and  5V  are  obtainable  from  any  one  sample  function. 
Most  of  the  tv  and  5V  are  zero.  It  is  only  when  we  have  an  iv  =  1 
and  iv+^±  1  that  tv>  0  and  =  1.  So  (17)  can  be  written: 


.  .  W.  +  W,  +  . 

xim  1  2 


.  +  W. 


E[return  time  J  just  left] 


(18) 


i.n  which  k  denotes  the  number  of  times  we  observe  the  system  leave 
state  1/  up  to  time  n,  and  W  is  the  mth  return  time.  Let  r  denote 
the  number  of  actual  transitions  (a  change  in  i)  that  occur  up  to 

tine  n.  Then  roughly  k  *  rn  =  no.  of  times  i  =  1;  ...rrr  =  no. 

JL  xv 

of  times  i  =  K.  Therefore  the  numerator  in  (18)  is  roughly 
rn2^2  +  rrr3^3  +  -rVK  an<*  s^nce  *  =  (18)  becomes 

B  [return  time  j  just  left  i=l] 


rr-,w~  +  . . .  +  nvwv 

- it_M - w - -  IS  ft,. 


TTi  Wj_ 


°Z  *  °3  _‘  *  *  +  qK 


w. 


W, 


(19) 


using  (12) .  For  any  state  i,  the  mean  recurrence  time  is 

1  ~  CTi  - 

Etreturn  time  |  just  left  i]  =  o.~  wi  •  (2°) 


12.  Our  final  example  concerns  the  observed  waiting  time  dis¬ 
tribution  for  a  given  state,  say  i  —  1.  When  the  steady-state 
condition  prevails,  we  observe  our  system,  and  given  that  i  -  1# 
we  inquire  when  it  arrived  in  i  =  1  and  when  does  it  leave,  and 
call  the  difference,  departure  time  minus  arrival  time,  the 
wait-time  (or  sojourn  time) .  The  distribution  of  this  variable 
is  not  the  same  as  the  given  wait  time  distribution.  We  denote 
this  variable  by  w,  and  we  calculate  its  characteristic  function 

E[ejaW]. 


Put 


F((u)  = 


r 


if  i0  =  1 
if  i 


0  £ 
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